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ay issue of BSD Magazine is dedicated to security matters 

with the use of Open Source solutions. On the following 
pages, you will find articles about Packet Filter, Jails and tools for 
troubleshooting, scanning, and text search. 


We start with Rob’s column, where he will discuss the Matter or 
property laws and how it happens that good solutions are beaten 
by technically less advanced ones and perish, 


Next, we announce the second release Of Spider-ool— the Too! 
for spidering web pages. Its author, Steve Micaller, Will explain its 
features, installation process, and simply NOW TL Works: 


In the Get Started section, Michael ShOWS Step py step Ov 10 
configure the firewall to only allow Spe@Ciic trailiG Tote ser Vice jails: 


This month’s Dev Corner covers PC-BSD ang WignignibasD: 
Kris will teach you more about jail management with Warden ana 
how to create jails via Hostname / Nickname and Change ana 
assign IP addresses on the fly. Meanwhile Lucas will introguce you 
to msearch — a full text search tool, that offers USErs The ability TO 
search against filenames or contents of text 7i/é@s. 


Then, Dru explores some of the third-party uulities Which are 
available to help you analyze the log and state table of a PF firewall. 


Next, we have the fourth part of Rob’s series on FreeBSD 
Programming Primer. This time, sysadmins have an opportunity 
to learn how to configure a development environment and write 
HTML, CSS, PHP, and SQL code. 


In May 2012, we published the article “Intro to Dtrace” by Carlos 
Antonio Neira, where he explained the system configuration to 
enable DTrace probes and some of this tool’s features. A year later, 
he comes back with a much deeper approach... 


We hope you will enjoy this issue and find many interesting 
articles! 


Patrycja Przybylowicz 
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Let’s Talk 


OGWhose ldea is it Anyway? 

By Rob Somerville 

With Apple fallen from grace as the world’s most 
valuable company, how can large technology-based 
companies succeed? The current trend for Intellectual 
Property laws can only increase the speed at which the 
race is towards the bottom... 


What’s New 


OSSpiderFoot 2.0: The Open Source 
Footprinting Tool 
By Steve Micallef 
The original version of SpiderFoot was created in 2005 
with the goal of being a freely available open source tool 
for footprinting an Internet domain name. Version 2.0 
was released May 2013 and is completely re-written in 
Python with loads of new functionality and is now highly 
extensible. The target user-base is penetration testers, 
system administrators and security enthusiasts who wish 
to gain a better understanding of what a domain name’s 
Internet footprint looks like. 


Get Started 


12 FreeBSD Jails Firewall with PF 
By Michael Shirk 

Features are available for fully virtualizing FreeBSD jail 
networking (as of FreeBSD 8.x). The code has improved 
in the current 9.x code base but to get a jail up and 
running with the current install, pf provides the necessary 
functionality to firewall off multiple jailed services. This 
article will cover basic jails configuration to highlight how 
to configure the firewall to only allow specific traffic to the 
service jails. 


Developer’s Corner 


16 Improvements to Jail Management via 
the Warden 
By Kris Moore 
Over the past few months, several exciting new features 
have been added to the Warden which greatly improve 
jail management on FreeBSD & PC-BSD systems.Now 
the Warden will be able to create jails via Hostname / 
Nickname, and change and assign IP addresses on the 
fly. This greatly simplifies jail creation via the command- 
line, allowing you to create the jail and then set addresses 
as needed later. 
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18S msearch: MidnightBSD Search 
By Lucas Holt 

MidnightBSD search, or msearch, is a full text search tool. 
It offers the user the ability to search against filenames 
or contents of text files. msearch is not meant to replace 
other tools like find, locate, or whereis. From this article 
you will learn the basic usage of the msearch tool and the 
reason why it was written. 


How To 


= OuUseful Utilities for PF 
By Dru Lavigne 

PF is a stateful firewall, meaning that it tracks the state of 
existing connections in a state table, allowing the firewall 
to quickly determine if packets are part of an established 
connection. PF also provides a logging facility and the 
firewall administrator controls which packets get logged by 
including the log keyword in only the firewall rules which 
should be logged when matched. This article explores 
some of the third-party utilities which are available to help 
you analyze the log and state table of a PF firewall. 


Admin 


= SFreeBSD Programming Primer — Part 4 
By Rob Somerville 

In the fourth part of our series on programming, we will 
continue to develop our CMS. Here we will examine how a 
modern CMS dynamically generates and controls content 
and implement a similar model in our PHP code. From 
this article you will learn how to configure a development 
environment and write HTML, CSS, PHP, and SQL code. 


Tips & Tricks 
> SDtTrace: A Deeper Approach 


By Carlos Antonio Neira 
The author of the article “Intro to DTrace”, published in 
May 2012 in BSD Magazine, has described DTrace all 
the way from configuring your system to enabling DTrace 
probes to the point of executing some D scripts to show 
you some DTrace features. This article will take a deeper 
approach on DTrace. 
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Whose Idea is it 
Anyway? 


With Apple fallen from grace as the world’s most valuable 
company, how can large technology-based companies succeed? 
The current trend for Intellectual Property laws can only increase 
the speed at which the race is towards the bottom. 


the most innovative user interface, developing a commitment from your 

customer base that is almost religious in its zeal would be enough, but 
no. The market — and the technology marketplace in particular — is fickle, yet the 
proponents of draconian Intellectual Property (IP) rights fail to grasp this fact. What 
is the latest de rigueur soon becomes passé as not only the technology evolves, but 
customer expectation rises. The paradox Is this: while it takes a tremendous amount 
of financial investment to develop new technology, the returns are often quite ran- 
dom and defy logic and statistical analysis. Take Betamax over VHS for example. ' 
Technologically VHS was not as advanced as Betamax, yet the underdog won the [ 
battle by having the support of the entertainment industry (partly due to the extra 
recording time VHS provided) and reaching the tipping point in the marketplace 
before Sony could roll out a 2 hour version. Result? The company that brought " 
the transistor radio and broadcast quality kit to the world was sorely under- 
mined by a more efficient but less innovative manufacturer. 

Now it could be argued that this is a strong basis for IP law, but the 
problem fundamentally remains — who has the right to an idea? Even 
more importantly, who has the right to lay sole claim to something that 
will bring major benefits to mankind? Throughout history there seems to 
be this “universal consciousness” where ideas arrive via the zeitgeist and 
monumental battles arise as to who has the best format, original concept, 
or design. Take Edison versus Tesla for example. Time and time again, the 
lone inventor is an endangered species when exposed to the power and force 
of the marketplace. Likewise, a multinational attempting to cling on to success 
based upon a single idea or philosophy is futile — yesterday’s success is no 
guarantee of tomorrow’s profitability. The success of the IBM PC was argu- 
ably not down to IBM's innovation, good design, or the fact that they were a 
market leader — it was the sweat shops in Asia producing clones untouched 
by Western patent law that blew the market right open. Of course, IBM hav- 
ing its fingers severely burned jumped on the IP bandwagon with Micro-chan- 
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nel, restricted developers by implementing a licensing pol- 
icy and guess what? MCA was dead in the water. Even 
Compag tried with Extended Industry Standard Architec- 
ture (EISA) but they could not overcome the juggernaut 
that the Industry Standard Architecture (ISA) had become. 
Let’s play devil’s advocate with the whole philosophy of 
IP. | am paid by my employer to write code, solve prob- 
lems, and innovate. Any ideas | come up with and any 
code | write belongs to my employer. That’s fair enough in 
a 9 to 5 environment. However, being the type of person 
that | am (incurable pedant), if my employer has a prob- 
lem or my code doesn't do what it says on the tin, | will 
worry about it. | will want to improve it. | am like a dog with 
a bone. And that means thinking about it — on the journey 
home, in the bath, when | wake up in the morning. My 
wife is witness to me sitting bolt upright in bed at 2:00 AM 
yelling “You need to compsurf that drive” before set- 

tling down to a more passive stage of uncon- 
sciousness. Now, | subconsciously solve 
the problem in a moment of revelation 
when | least expect it at 4:00 AM. 
Who has the intellectual property 
on that? According to the law- 
yers, | am supposed to chal- 
lenge my employer and say 
it was my idea but as it was 
outside of my contractual 
hours, | cannot share it 
with them. Or maybe 
not. The suggestion is 
ludicrous, — unethical, 

and prohibitive, yet this 

is where IP is driving 

the innovators and the 
creatives. | understand 

the dilemma that is at 
the heart of IP — reward 
and recognition. A good 
workman is worth his 
wages, and credit where 
credit is due. How can we 
restore the value of the in- 
novators, those that suc- 
cessfully think outside the 
box, in a society where 
everybody is a winner? 
How can large organiza- 
tions profit yet at the same 
time protect their invest- 
ment? Certainly the digital 
age brings huge challenges 
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in this regard. It takes little cost or effort to copy software, 
a customer database, or in the case of Wikileaks, state 
secrets. We live in an age where technology is demolish- 
ing all the boundaries and traditional rules of ethics and 
conceivably the universe. | cannot clone a car in the time 
it would take me to clone a credit card, yet potentially the 
amount of profit | could make from this (albeit illegally) is 
potentially more that the value of a car that would take 
one individual months — if not years — to replicate. What is 
valuable now — information and power — hasn't changed, 
but the medium and how it is delivered and extracted has. 

The last time we had a technological revolution on such 
a scale, we were living in the 1400's. It could be reason- 
ably argued that the Protestant Reformation was a direct 
consequence of Johannes Gutenberg and the printing 
press. The established rule crumbled, and the renais- 
sance brought enlightenment and a much needed free- 
dom of information exchange. Part of the reason for this 
explosion in knowledge was ironically due to the way in- 
formation was disseminated prior to the black death — 
monks in monasteries were responsible for producing 
books, and the church was anxious to control what was 
acceptable. The plague reduced the ability to produce 
books efficiently, and from an economist’s point of view, 
the printing press filled that market need. 

Large organizations, like large groups of people — don't 
like change. The flexibility of the small or medium sized 
company far outweighs that of the established behe- 
moths. All large technology companies must face the fact 
that they are not immortal or omnipotent, as history proves 
time and again. It’s that fickle marketplace again. Red- 
hat has made major inroads into powering major finan- 
cial institutions, yet its share price remains a fraction of 
Apple Inc. The fact that a business model based on Open 
Source can breach the bulwark of the capitalist business 
model should be a wake up call to those that believe that 
the traditional rules still apply. Technology makes a great 
slave but a terrible master. We live in interesting times. 


ROB SOMERVILLE 

Rob Somerville has been passionate about technology since 
his early teens. A keen advocate of open systems since the mid 
eighties, he has worked in many corporate sectors including fi- 
nance, automotive, airlines, government and media in a vari- 
ety of roles from technical support, system administrator, de- 
veloper, systems integrator and IT manager. He has moved on 
from CP/M and nixie tubes but keeps a soldering iron handy 
just in case. 
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WHAT’S NEW 


SpiderFoot 2.0 


The Open Source Footprinting Tool 


The original version of SpiderFoot was created in 2005 with the 
goal of being a freely available open source tool for footprinting an 
Internet domain name. Back then, it was written in C# and only ran 
on the Windows platform with fairly limited functionality. Version 
2.0 was released May 2013 and is completely re-written in Python 
with loads of new functionality and is now highly extensible. 


What you will learn... 

« What is footprinting, and why is it used? 

« What does SpiderFoot do, and how can it be of use to you? 
¢ How to install and use SpiderFoot 


administrators and security enthusiasts who wish to 

gain a better understanding of what a domain name's 
Internet footprint looks like, and perhaps where there may be 
undesirable information leakage from that domain. 


yT he target user-base is penetration testers, system 


What is Footprinting? 
In a generic sense, footprinting is the process of under- 
standing as much as possible about an entity. In the con- 
text of the Internet and specifically SpiderFoot, that entity 
is a DNS domain name, for instance, Google.com. Some 
people interpret footprinting as port scanning, others as 
spidering web pages and so on, but what constitutes a 
“complete” footprint is completely open and can actually 
change over time. 

lf you consider what the Internet looked like in the year 
2000, the footprint of an Internet domain name would 
have included hostnames/sub-domains, IP addresses, 
open ports, and others, but it would not have included 
anything about social media presence. In the same vein, 
the Internet is continually evolving with the addition of rich 
data sources that provide a wealth of information about In- 
ternet entities that were not available previously or only of- 
fered in unstructured form. A lot of that has since changed, 
not only resulting in more widely available data, but also 
data available as web services, thus making its collection 
and analysis more automatable. 
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What you should know... 

¢ A basic understanding of TCP/IP and how the Internet works would 
help, but is not really essential. 

« If you're using SpiderFoot on Linux or *BSD, basic knowledge of Py- 
thon might help 


How is it Done? 

The most basic data source for footprinting is the website 
of the entity itself. Simple things like e-mail addresses, 
hostnames/sub-domains, web server versions, web serv- 
er technologies, and much more can be gathered simply 
by fetching web pages from the target, following links, per- 
forming some regular expression checks, and analysing 
HTTP headers. 

But the real power of footprinting is combining data from 
one activity with another to come up with a bigger picture. A 
simple example is performing a DNS lookup of the entity’s 
domain name to get the IP address, then looking up the 
IP address in an Internet address registrar (for example, 
RIPE, ARIN or APNIC) and from there, determining wheth- 
er the entity owns the entire network range that the IP re- 
sides on. Then, armed with that information, you can port 
scan, banner grab, and so on in order to add to your foot- 
print and in turn use the information obtained there (host- 
names, software versions, and other data mentioned in 
connection banners is one example), to build it up further. 


Why Footprint? 

Footprinting is not an academic exercise; it is typically the 
precursor to a penetration test, enabling the penetration 
tester to gain a birds-eye view into what an entity really 
looks like at a technical level, what the entry points may 
be for the penetration test, dependencies to other entities 
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(ISPs and Hosting providers, for example), and also po- 
tential early indicators of points of weakness. Additional- 
ly, many large organizations struggle with managing their 
network perimeter and having an outside-in view of what 
an entity looks like can help gain and maintain visibility. 


SpiderFoot 

Now that you understand what footprinting is, how it’s do- 
ne and why, it’s more meaningful when we say that Spi- 
derFoot is a footprinting tool designed to automate the 
footprinting process to the fullest extent possible by ex- 
tracting information from whatever data can be obtained 
freely from the Internet. 


Background 

When SpiderFoot vO.1b was originally released in 2005, it 
used the then-available Google API, Netcraft and website 
spidering as methods for building up a footprint, and these 
methods were hard-coded into the tool. Despite Google 
dropping support for its API and Netcraft blocking access 
to much of its data, SpiderFoot continued to be download- 
ed and used - clearly a need still existed for automated 
footprinting. 


Modules 

In version 2.0, which is completely modular and entirely 
re-written in Python, each method for building up the foot- 
print is encapsulated in its own module. In addition, mod- 
ules generate each data element identified (i.e. an IP ad- 
dress, a web page, etc.) as an “event” that is consumed 
by other modules listening for that event. This model en- 
ables SpiderFoot to extract “maximum value” out of each 
piece of data found. SpiderFoot’s modules, at the time of 
writing, are as follows: 


¢* sfp_ dns: Performs a number of DNS checks to ob- 
tain IP Addresses and Affiliates. 

* sfp  geoip: Identifies the physical location of IP ad- 
dresses identified. 

* sfp googlesearch: Some light Google scraping to 
identify links for spidering. 

* sfp mail: Identify e-mail addresses in any obtained 
web content. 

* sfp pageinfo: Information about web pages (do they 
take passwords, do they contain forms, etc.) 

* efp - portscan basic. Scans for commonly open 
TCP ports on IP addresses found. 

* sfp ripe: Queries RIPE to identify owned netblocks 
and other info. 

* sfp similar: Searches various sources to identify 
similar looking domain names. 


www.bsdmag.org 


* sfp spider: Spidering of web-pages to extract con- 
tent for searching. Probably the most valuable module. 

* sfp stor db: Stores scan results into the back-end 
SpiderFoot database. This is modularized for future 
scalability purposes. For now it stores results to an in- 
ternal SQLite database. 

* sfp subdomain: Identify hostnames / sub-domain 
names in URLs and obtained content. 

* sfp_websvr: Obtain web server banners to identify 
versions of web servers and related technology being 
used. 

* sfp xref: Identify whether other domains are associ- 
ates (“Affiliates”) of the target. 


Going into the inner workings of each module is beyond the 
scope of this article, but you can find the source code to 
each of them and more at the GitHub link provided below. 


Installing 
On Linux, *~BSD or Solaris, installing and running Spider- 
Foot should be a breeze. Provided you have Python 2.6 or 
2./ (Python 3.x support coming soon), all you'll need are 
CherryPy and Mako, two modules SpiderFoot uses for its 
web-based interface. 

lam using FreeBSD 9.1-RELEASE as an example here, 
but if you’re using another BSD, you'll probably need to 
adapt your approach slightly. If you’re using Linux, follow 
the instructions in the rEapme file included in the Spider- 
Foot package. 


Step 1 
Install pip if you don’t have it already. This will enable you 
to easily install Python packages. 


# cd /usr/ports/devel/py-pip 


# make && make install 


Step 2 
Install SQLite for Python. 


# cd /usr/ports/databases/py-sqlite3 


# make && make install 


Step 3 
Install CherryPy and Mako Python modules. 


# pip install cherrypy 
# pip install mako 


Step 4 
Unpack SpiderFoot into a location of your choice. 
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“5 tar Z2xt Spiderroot—Z2.x.x-Sre.tar.¢gz 


~S cd spiderfoot 


Starting 
To run SpiderFoot, simply execute sf.py from the direc- 
tory you extracted SpiderFoot into: 


S python ./Sf.py 


Once executed, a web-server will be started, which 
by default will listen on 127.0.0.1:5001. You can then 
use the web-browser of your choice by browsing to 
http://127.0.0.1:5001. You should then see something like 
this: Figure 1. 


Configuring 

With the exception of the IP and Port bound to by the Spi- 
derFoot web server, which are set on the command-line, 
all other SpiderFoot configuration settings are controlled 
in the UI. After clicking on the Settings button in the title 
bar, you will be presented with a few global settings fol- 
lowed by module-specific settings (Figure 2). 

Here you can configure things like the User-Agent string 
to use during spidering, the period of time to pause be- 
tween web requests, TCP ports to scan, and more. Save 
settings keep them persistent between scans even if you 
stop and start SpiderFoot completely. 


SpiderFoot +! Bios - OA 


Scans 


Ne scan hishory 


There & CUneniy ino heen of pevicly Tun scans Pees click Pie 


Figure 1. The SpiderFoot interface after starting it up for the first time 
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Modula Option Value 
1p_ dred Carmen suber £6 ry ae 
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Figure 2. User interface for setting SpiderFoot’s configuration 
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Running Scans 

Running a scan is extremely simple — click the New Scan 
button in the title bar, then give the scan a descriptive name, 
specify the target you want to scan, and then select which 
modules you would like enabled or disabled: Figure 3. 


Browsing Results 

Thanks to the introduction of an SQLite database back- 
end in 2.0, scan results are stored — in real time as the 
scan progresses — locally in a database file. By clicking 
on the Scans button in the title bar, you can see a list of 
scans run previously, in addition to the scan you have just 
initiated. Click the name of the scan you are interested in 
and you will be presented with the data available for that 
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Figure 4. A list of data elements making up the footprint of a target 
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On the Web 


http://www.spiderfoot.net — The SpiderFoot website. 
http://github.com/smicallef/spiderfoot — SpiderFoot source 
code on GitHub. 

http://twitter.com/binarypool — SpiderFoot twitter feed. 
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Figure 5. A detailed listing of the data elements (in this case, Web 
Servers) from a footprint 


scan. This starts getting populated the moment a scan ini- 
tiates; see Figure 4. From here you are then able to “drill 
down’ into the actual data. Data can also be exported to 
CSV format for offline manipulation/analysis if desired by 
clicking the blue icon to the right (Figure 5). 


Looking Ahead 

Hopefully this article has given some insight into the in- 
teresting world of footprinting with SpiderFoot. The tool is 
still very much in its infancy, but it does the job it is tasked 
to do with big plans for new modules and additional core 
functionality. Plans for future modules include SSL certifi- 
cate checks, identifying the entity’s ISPs (possibly using 
Traceroute or BGP tables), and 3rd party integration with 
vulnerability scanners and the like, but you can get a full 
list on the GitHub project site with the link provided below. 


Happy Fooiprinting! 


STEVE MICALLEF 

Steve Micallef has been specializing in IT Security for the past 13 
years, currently working in a large financial institution. With a 
passion for security and for delivering quality security solutions, 
Steve has designed, built and delivered global solutions in the 
areas of SIEM (Security Information & Event Management), Vul- 
nerability Scanning, Data Leakage Prevention and more. 

Steve created SpiderFoot with the goal of giving Penetration 
Testers a way to automate the more cumbersome and time-con- 
suming process of a penetration test - footprinting. He is con- 
stantly looking at ways to improve the tool, always with that 
goal in mind. 
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GET STARTED 


with PF 


FreeBSD Jails Firewall 


Features are available for fully virtualizing FreeBSD jail 
networking (as of FreeBSD 8.x). The code has improved 

in the current 9.x code base but to get a jail up and 
running with the current install, PF provides the necessary 
functionality to firewall off multiple jailed services. 


What you will learn... 
¢ Configuration of PF to setup nat and rdr rules for ssh access 
- Basic setup of jails using ezjail and the jls and jexec utils 


highlight how to configure the firewall to only al- 

low specific traffic to the service jails. The first thing 
that needs to be completed is an install of FreeBSD 9.1 
(amd64) with an install of the system source and the ports 
tree (See FREEBSD-INSTALL for installation instruc- 
tions). To help with the jail configuration, | am using ezjail. 
Listing 1 shows how to install the ezjai1 port and how to 
configure a basic jail called “ssh-test”. 


yT his article will cover basic jails configuration to 


What you should know... 
¢ Basic FreeBSD knowledge to navigate the command line 
¢ Familiarity with PF and navigating the ports system 


The key thing about this configuration is that | am using an 
IP on the local interface “127.0.1.1". ezjail-admin will output 
that the interface has not been configured when creating the 
jail. Listing 2 demonstrates the configuration to get the jail up 
and running on the local interface with an alias on 100. 

Once the system has rebooted, the new jail will be up 
and running with the local alias IP “127.0.1.1”. Listing 3 
shows the output of the ;1s command and the alias on the 
local interface. 


Listing 1. /nstall ezjail and setup ssh-test jail. (Note: for the jail creation, em0 is the interface type for a VirtualBox VM. This may be different in 


your setup so use the appropriate interface) 


# Ce /Uei/ pores, sysutrils/ezjail/ 
# make -DBATCH install clean 


(Output from install ezjail port) 
# echo ‘ezjail enable="YES"’ >> /etc/rc.conf 


# e@Zjail-admin install 


(This will take some time, as it creates a base jail) 


# ezjail-admin create ssh-test ‘em0|127.0.1.1’ 


Listing 2. Configuring the interface to load up with the jail IP 


# cone, “reomic Nolmeltasi— inner 127.05 Moab netmask OxtiPiniti@ >> ene) ne. cont 
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FreeBSD Jails Firewall with PF 


For now, we will setup ssh to automatically start in- 
side the jail. In addition, we will create a “test-user” to be 
able to login over ssh. Listing 4 shows the commands to 
change the default ssh port to 2022, add the test-user and 
enable sshd on startup. 

At this point, when running from the host operating sys- 
tem, you should be able to ssh on port 2022 into the jail. 
However, if you wanted to connect in from a remote sys- 
tem, the local interface connection would not be available. 
This is where pt can be configured to redirect traffic into 


the jail. Listing 5 shows a basic pf configuration to provide 
NAT redirection for the jail. 

The firewall rules essentially take all tcp port 2022 traf- 
fic and redirect it to the jailed sshd service. Any traffic sent 
back will be NATed on the host interface (em0 in this ex- 
ample). The firewall needs to be configured at startup, 
which is demonstrated in Listing 6. 

The system will reboot and from another remote system 
(or the host OS) you should be able to ssh on port 2022 
into the jail. Check the above configuration settings if this 


used with the jexec command to run a shell inside the jail) 


# jls 
JID IP Address 
ey Oe ee 


Hostname 
ssh-test 
# ifconfig 100 


inet6o ::1 prefixlen 128 

inet6o fe80::1%1lo0 prefixlen 64 scopeid 0x5 
nec 127 0.021 netmask Oxi r 000000 

mer. L270. lh nerndask Uxirr iinet 


ndo options=21<PERFORMNUD,AUTO LINKLOCAL> 


# jexec 1 tcsh 
COOkCsch=rest-/ + ls 
ses lade COPYRIGHT loukial dev ils 


.profile boot etc libexec 


Coondcehi—teost 7 4 


basejail 


TOOL@SSh-—test:/ ¢# Sed =1 *” 


root@ssh-test:/ # passwd test-user 
Changing local password for test-user 
New Password: 

Retype New Password: 

HOOECSSn=test:/ 7 /euc/re.d/ sshd Start 


(Output from the SSH key generation) 
root@ssh-test:/ # sockstat -4 


Listing 3. Output of jls showing new interface alias and the ssh-test jail up and running. (Note: the jail ID in this configuration is 1, which is 


Pah 
/usr/jails/ssh-test 


lo0: flags=8049<UP, LOOPBACK, RUNNING,MULTICAST> metric 0 mtu 16384 
options=600003<RXCSUM, TXCSUM,RXCSUM IPV6,TXCSUM IPVo> 


Listing 4. Changing the default port for sshd and enable it on startup for the jail. 
‘e/g wee 22s Percic 2022 ef eine/eclay elmel ceiming 


root@ssh-test:/ # echo ‘sshd enable="YES”’ >> /etc/rc.conf 


root@ssh-test:/ # pw user add -n test-user -s /bin/csh -m 


media iSiaore iOCAE sys usr 


mnt rescue Soon tmp var 


USER COMMAND Ep FD PROTO LOCAL ADDRESS FOREIGN ADDRESS 
BOOt sshd 130d 3 teo4 AES OA NES Ie «~* 

OVO NC sendmail 114i 3° srep4 Zens es ee 2 top sal 

OOIE Sys loge 1087 6 udp A Ola ties eae = 
root@ssh-test:/ # exit 

# 


(You should be out of the jail for the next steps.) 
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GET STARTED 


Listing 5. /etc/pf.conf configuration for redirecting remote traffic 
to jailed services. (Note: adjust the Sext_if according to the interface 
type you are using.) 


WT 


ext 1f£="emd 


Sol eS P= 0 


# nat all jail traffic 
ENE (2 Clal Srepcie Ie sero SIS SRUNE SIE ico) clin =e ((Seotc a8) 


# port 2022 is redirected to the jail 
rot pass Of Sexe Ie prove tcp Erom amy LO any Pore 20722 
SoS SCOH ESI oome 2022 


f pote 22 on hose 

Pass 1m loc om exe yur CE puero bcp EEOM any LOnpore 
22 tags —S/ ok 

pass out log on Sext if proto tcp all keep state flags S/ 
SA 


Listing 6. /etc/rc.conf configuration for enabling pf on startup. 


# echo ‘pf enable="YES”’ >> /etc/rc.conf 


ie 


# echo ‘pf rules="/etc/pf.conf”’ >> /etc/rce.conf 


te 


Schon pr prOgrel=— / sbanm/ pier” >> enc, mc, com 


#f 
#7 echo “pr tlags—"“" >> /etc/re.cont 
# 


reboot 


Listing 7. Running ssh to remotely connect into the jail. 


Sse -p2022 test —uce rd oZ ho d7 58.20 

Passwords 

Kase legun: Mone May io). 247045 200s from 92-1868 50.1 

FreeBSD 9.1-RELEASE (GENERIC) #0 1243825: Tue Dec 4 
Oe 3 Oe Ume e70Nk7 


Welcome to FreeBSD! 


Before seeking technical support, please use the 


following resources: 


Oo Security advisories and updated errata information 
for all releases are 
ele Imes / / Vin. Mieeel SI). Orc) eleeises/ — ellliveiies) coins lic 
the ERRATA section 


for your release first as it’s updated frequently. 


o The Handbook and FAQ documents are at http://www. 
FreeBSD.org/ and, 
along with the mailing lists, can be searched by 
Jong EC 
eeoe / / inn MeSSesI Cig) SScucl . Ie elie Coe joackeage 
has been installed 
(Ob eveheds foo ce odds—aelengSebechod doc swieke 
lang 1s the 
2-letter language code, e.g. en), they are also 
available formatted 


in) /iusi/ local/share/doc/ freebsd. 


If you still have a question or problem, please take the 
Owicjethe Oic 

“uname -a’, along with any relevant error messages, and 
email it 

as a question to the questions@FreeBSD.org mailing list. 
If you are 

unfamiliar with FreeBSD’s directory layout, please refer 
to the hier (7) 

manual page. 


If you are not familiar with manual pages, 


type “man man’. 


Edit /etc/motd to change this login announcement. 


Q 


test-user@ssh-test:/home/test-user % 


is not working. Listing 7 shows the output of remotely log- 
ging into the jail with ssh. This is only a basic configuration 
for providing services within a jail. If you include the ports 
system for the jail, additional software can be added to 
provide web services and any other basic services. Using 


References 
¢ FREEBSD-INSTALL: http:/;www.freebsd.org/doc/handbook/ 
bsdinstall. html 


« Jails: http://www.freebsd.org/doc/handbook/ails.html 
¢  ezjail: http://erdgeist.org/arts/software/ezjail/ 
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pf, all services can be provided to external connections 
while at the same time authorizing only the necessary 
ports for jail access giving additional security for services. 


MICHAEL SHIRK 

Michael Shirk is a BSD zealot who has worked with OpenBSD and 
FreeBSD for over 7 years. He works in the security community and 
supports Open Source security products that run on BSD operat- 
ing systems. Michael is the Chief Executive Manager of Daemon 
Security Inc., a company which provides security solutions utiliz- 
ing the BSD operating systems: http://www.daemon-security.com 


05/2013 


BSDCAN 2013 


THE BEST EVENT OF 2013 
http://www.bsdcan.org/ 


Ottawa, Canada 


BSDCan 2013 — The event to be at this year 


BSDCAN 2013 


WHERE 


15-16 May — tutorials 
17-18 May — conference 


WHO 


All who are working on and with 4.4BSD 
based operating systems and related 
projects. 


VENUE 


University of Ottawa 
http://www.uottawa.ca/ 


High value. Low cost. Something for everyone. 


AT FEES YOU CAN AFFORD 


We plan to keep to a minimum. As such, the 
conference will be held at University of Ottawa 
and accommodation is available within the 
University residences. Hotels are also within 
close walking distance of the conference 
venue. 


WHAT DOES IT COST? 


Type CAD 
Individual $195 
Corporate $350 
Additional Corporate $175 
Student $60 
Tutorial (per half day) $60 


University of Ottawa Staff&Student $45 


Take the BSDA Certification exam. 
For details see 
http://bsdcertification.org/ 


¢ ANetBSD based Tracking Radar 

¢ FreeBSD Kernel Security 

¢ Automating the deployment of FreeBSD 
& PC-BSD 

¢ Backup and Restore with Bacula 

¢ Benchmarking FreeBSD 

¢ Switching from Linux to FreeBSD 

¢ DNSSec: Troubleshooting and 
Deployment 

¢ Embedding NetBSD: VOIP applications 

¢ FreeBSD, Capsicum, GELI and ZFS 

¢ FreeBSD Doc Sprint 

¢ FreeBSD storage options 

¢ Hands-on bhyve, the BSD Hypervi 

¢ — Introduction to pkgsrc 
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Warden® 


Improvements to Jail 
Management via the 


Over the past few months, several exciting new features have been 
added to the Warden which greatly improve jail management on 


FreeBSD & PC-BSD systems. 


PC 


. istorically the Warden has always organized its 


collections of jails via a primary IP address. This 

was functional but not the optimal point of refer- 
ence when dealing with large quantities of jails on a sys- 
tem. Thanks to some recent cooperation between the PC- 
BSD & FreeNAS teams, this has been done away with 
and improved. 

Now the Warden will be able to create jails via Host- 
name / Nickname, and change and assign IP addresses 
on the fly. This greatly simplifies jail creation via the com- 
mand-line, allowing you to create the jail and then set ad- 
dresses as needed later. 


# warden create myjail 
# warden set ipv4 myjail 192.168.0.25/24 
# warden set ipv6 myjail fe80::8a89:a5ff:fe52:ad19 


In addition to being able to set both a primary lpv4 and 
lpv6 address, jails can also include a number of oth- 
er addresses. Any number of aliases for both lpv4 and 
lpv6 can be set, along with the default router for lpv4 & 
lpv6. The Warden is also now configured to automati- 
cally use the VNET option, giving each jail its own virtu- 
al network stack. This includes giving jails their own net- 
work interface and can allow a wider variety of services 
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to run behind a jailed interface. Because of this feature, 
the Warden will require that your kernel is compiled with 
the vimace option enabled. Users of PC-BSD & TrueOS 
rolling-release will be able to update to this kernel via the 
normal freebsd-update mechanisms. With these new 
features brings new options which can be set via the 
command-line: 


# warden set myjail alias-ipv4 192.168.0.200/24 
# warden set myjail bridge-ipv4 192.168.0.2/24 
# warden set myjail alias-bridge-ipv4 192.168.0.3/24 


Along with new virtual networking functionality, the War- 
den also has a few new tricks up its sleeve. For PC-BSD 
& TrueOS 9.1 and higher users, we have begun build- 
ing and maintaining our own full package repository us- 
ing pkgng. When creating standard jails, the Warden will 
handle automatically boot-strapping the pkgng package 
and repository. 

Should this process be unable to complete, such as 
on a system with no internet connectivity, or be cor- 
rupted by a well-meaning end user, it can be re-run at 
any time: 


# warden bspkgng myjail 
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Improvements to Jail Management via the Warden 


Another long-requested feature was the ability for the 
Warden to manage setting various permissions and flags 
for a jail and handle user-supplied nullfs mounts. These 
can both be easily configured per-jail by using the “set 
flags” and “fstab” options respectively. 


jeworoen Sce Mylemeemyjail allow.raw sockets=true 


# warden fstab myjail 


All of these new features and options are also fully ex- 
portable. This will allow you the ability to provision a jail 
on your PC-BSD workstation, either via the command- 


Jail Configuration x) Way) st 


Jail Configuration 


‘IPv4 | 1PV6 | Aliases || Permissions. 
| (| IPv4 Address 

127.0.0.7/24 
| IPv4 Bridge Address 


IFv4 Default Router 


Save Cancel 


Figure 1. The jails [Pv4 configuration 


Jail Configuration 


(< 
> 
(x 


Jail Configuration 
|'Pv4 |/1Pv6 || Aliases | Permissions | 


The following permissions can be enabled or diabled for this jail. For | 
more details hover over each item or refer to the jail manpage. 


|) allow.set_hostname 
allow.sysvipc 

¥) allow.raw_sockets 
allow.chflags 
allow.mount 
allow.mount.devfs 
allow.mount.nullfs 
allow.mount.procfs 
allow.mount.zfs 
allow.quotas 

| allow.socket_af 


Save Cancel 


Figure 2. Setting jail permissions 


www.bsdmag.org 


line or GUI. Once you have finished the initial configura- 
tion and testing of your jail, you can then easily export it 
to a single archive file. This export file can then be taken 
to another system, such as FreeNAS, and then imported. 


# warden export myjail -—dir=/exports 


# warden import /exports/myjail.wdn 


At the time of this writing many of these changes are al- 
so being implemented into the Warden’s Graphical In- 
terface. As easy as the command-line flags may be, the 
GUI takes it a step further, making jail creation and man- 
agement possible without having to remember or look up 
a single command. 

So what is next for the Warden? Even with these new 
features still hot off the press, there are other improve- 
ments waiting in the wings. One of these will be the abil- 
ity to create and manage various jail “templates”. This will 
allow you to build a jail template for a particular FreeBSD 
release (Say you have a product which needs to run on 
8.3). By creating the 8.3 template, you will be able to cus- 
tomize it with software or configuration options specific to 
your needs. Then when it comes time to build jails, you 
will be given the option of using the latest release or your 
own jail template. Stay tuned to BSD Magazine for more 
details on this in a future issue. 


KRIS MOORE 

Kris Moore is the founder and lead developer of PC-BSD. He lives 
with his wife and four children in East Tennessee, USA and enjoys 
building custom PC’s and gaming in his (limited) spare time. He 
can be reached at: kris@pcbsd.org. 
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msearch: MidnightBSD 


Search 


A few years ago, | was trying to find a file on my 
MidnightBSD desktop system. | couldn’t remember the 
name, but knew there was a specific phrase in it. | could use 
the grep command to find the file, but it would take time. 


What you will learn... 
« the history of the msearch tool and why it was written, 
- basic usage of the msearch tool 


Mac OS X. | also considered how terrible most open 
source full text search engines operate. | decided to 
write my own search tool to make searching for files easier. 


te about how quickly Apple’s Spotlight works in 


Using MidnightBSD Search 
MidnightBSD search, or msearch is a full text search tool. 
It offers the user the ability to search against filenames or 
contents of text files. 

msearch is not meant to replace other tools like find, lo- 
cate or whereis. 


Table 1. msearch option flags 


-C Print the match count only. 

-|<number> — Limit the number of results 

-r Print the ranking information with full text results 

-t Perform a full text search rather than just using 
filenames 

-Z Print pathnames separated by an ASCII NUL 


character rather than a newline. 


How Does MidnightBSD Search Work? 

Files on the system are indexed weekly from a period- 
ic script that runs an indexing program. The indexes are 
used by the command line tool when executing searches. 
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What you should know... 


« how to install MidnightBSD or download a virtual machine image 


Indexing in action 

msearch.index indexes files on the system by determin- 
ing if the file is a text file using libmagic, reading the first 
20KB of the file and loading it into the full text indexer. The 
results are stored in SQLite databases; they are stored in 


J viele) dey meeamelie 


Listing 1. Example search queries 
# Filename based search, limited to 10 results. 


msearch -l 10 msearch 


usr/bin/msearch 
usr/include/msearch.h 
usr/lib/libmsearch.a 
usr/lib/libmsearch p.a 


usr /iab/ li bmsearcae so | 


/ 
/ 
/ 
/ 
/ 
/ 


ict? iti /ieomeeacemece 


/usr/libexec/msearch. index 

# Text based search 

Mecane het heads Holt” 

/usr/local/mailman/archives/public/midnightbsd- 
users/2007-August.txt 

/usr/local/mailman/archives/public/midnightbsd- 
users/2011-February.txt 

/usr/local/mailman/archives/public/midnightbsd- 

kernel/2008-September. txt 
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On the Web 
http://www.midnightbsd.org/-— MidnightBSD Project website, 
http://www.midnightbsd.org/cgi-bin/cvsweb.cgi/src/lib/libm- 
search/ — msearch library. 


Glossary 
msearch 
sqlite 


The msearch.db file contains a list of filenames, own- 
ership information, sizes, and other general metadata. 
msearch_full.db contains the full text search data. 


Turn on msearch indexing 
Indexing is enabled by adding meekly msearch 
enable="YES” tO /etc/periodic.conf. If you have many 
files, it is recommended to have at least a few gigabytes 
of free space on the /var mount point. 

Once the index has been generated for the first time, 
you will be able to use msearch to find files. 


Extending MidnightBSD Search 
msearch is built on top of a shared library, libmsearch, 
that allows developers to integrate search functionality in- 
to their own applications. Functions for creating and ma- 
nipulating indexes, as well as performing searches are in- 
cluded. 

Consult the msearch.h header file for a complete list of 
functions. 


Future Directions 

Following the 0.4-RELEASE of MidnightBSD, | plan to 
write a graphical application to extend searching and a 
new indexer. Scalability is a concern with regard to in- 
dex storage size. Creating an indexing daemon would al- 
low the index to maintain fresh. This would require use of 
kqueue or porting inotify from Linux. 


Summary 
msearch is an easy to use full text search tool for Mid- 
nightBSD. It allows users to quickly search text files on 
their system. 


LUCAS HOLT 

Lucas Holt is the founder of the MidnightBSD project and a Se- 
nior Application Programmer/Analyst for the University of Mich- 
igan in Ann Arbor, MI, USA. 
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The BSD Certification Group Inc. 
(BSDCG) is a non-profit organization 
committed to creating and 
maintaining a global certification 
standard for system administration 
on BSD based operating systems. 


@ WHAT CERTIFICATIONS ARE AVAILABLE? 


BSDA: Entry-level certification suited for candidates 
with a general Unix background and at least six months of 
experience with BSD systems. 


BSDP: Advanced certification for senior system administrators 
with at least three years of experience on BSD systems. 
Successful BSDP candidates are able to demonstrate 

strong to expert skills in BSD Unix system administration. 


@ WHERE CAN I GET CERTIFIED? 


We're pleased to announce that after 7 months of 
negotiations and the work required to make the exam 
available in a computer based format, that the BSDA 
exam is now available at several hundred testing centers 
around the world. Paper based BSDA exams cost $75 USD. 
Computer based BSDA exams cost $150 USD. The price of 
the BSDP exams are yet to be determined. 


Payments are made through our registration website: 
https://register.6sdcertification.org//register/payment 


@_ WHERE CAN 1 GET MORE INFORMATION? 


More information and links to our mailing lists, LinkedIn 
groups, and Facebook group are available at our website: 
http://www.bsdcertification.org 


Registration for upcoming exam events is available at our 
registration website: 
https://register.bsdcertification.org//register/get-a-bsdcg-id 
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Useful Utilities for PF 


This article explores some of the third-party utilities which 
are available to help you analyze the log and state table of a 


PF firewall. 


What you will learn... 
« How to view the PF state table in real time 
« How to convert the PF log to HTML format or graph format 


ect. PF has also been ported to FreeBSD, NetBSD, 

and DragonFly BSD. You can learn more about PF 
and its features in the PF User’s Guide at http:/Avww. 
openbsd.org/faq/pf/. 

PF is a stateful firewall, meaning that it tracks the state 
of existing connections in a state table, allowing the fire- 
wall to quickly determine if packets are part of an es- 
tablished connection. PF also provides a logging facility 
and the firewall administrator controls which packets get 
logged by including the log keyword in only the firewall 
rules which should be logged when matched. 

PF provides the pfct1 utility for displaying the state ta- 
ble and the built-in tcpdump utility can be used to view the 
PF log. In addition to these tools, some third-party pack- 
ages can be installed on BSD systems. These can be 
used to manipulate information from the state table and 
the PF logging facility in order to get a different view on 
what is happening with the firewall. This article provides 


T he PF firewall is developed by the OpenBSD Proj- 


What you should know... 
- How to restart PF 
¢ How to install third-party software on your BSD system 


an overview of the following utilities: pftop, pflogx, and 
pfstat. These utilities were tested on a PC-BSD system 
and the utilities were installed using FreeBSD packages. 
This article assumes that you already know how to restart 
PF and how to install software on your BSD system using 
packages, ports, or pkgsrc. 


pftop 


¢ Website: htto://www.eee.metu.edu.tr/~canacar/pftop/ 

¢ Availability: pkgsrc, FreeBSD and OpenBSD packages 

¢ Description: provides real time display of PF state ta- 
ble and rule statistics 


This utility is similar to top as it provides a real time, co- 
lumnar display. However, instead of displaying the top 
processes running on the system, it displays real time 
information about the current connections in the PF 
state table. 


Listing 1. pfctl View of State Table 


jeueeic.l =e) sreciuss 
GulieeeOu tle = hoes Ma eA eae oe ene 4 eA a 


ula itrches 10S), ullicrow gl ile ors) 
(smiperest Of CUEDUE...-) 


=, 22450022512 5353 


ESTABLISHED: ESTABLISHED 
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Figure 1. Default pftop Display 


Figure 2. Viewing Loaded Rules Using pfctl 
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Typically, the state table is read using pfct1 as seen in 
the following example. This output is from a PC-BSD sys- 
tem that is downloading a PBI using AppCafe. 

Figure 1 shows the same state table. This time, the dis- 
play is generated by typing pftop. 

In order, the columns in this default view list the proto- 
col (TCP or UDP), the direction (into the system or out of 
the system), the source address and socket, the destina- 
tion address and port, the state of the connection, the age 
of the connection, how long until that connection expires 
from the state table, the number of packets in that connec- 
tion, and the number of bytes transferred. 

pftop also provides a view for displaying which rules are 
currently loaded. First, Figure 2 shows which firewall rules 
have been loaded using the built-in pfct1. 

Next, Figure 3 shows the same rules, this time viewed 
USINg pftop. This display adds information such as the 
number of packets, bytes, and established connections 
(states) associated with each rule. 

pftop also provides an interactive mode where key- 
strokes can be used to modify the view, sort the column 
order, change the number of lines to display, and to pause 
or restart the display. Display filters can also be created 
USING tcpdump syntax. Refer to pftop(8) for details. 


¢ Website: http://akldev.free.fr/pflogx/ 

¢ Availability: FreeBSD and OpenBSD packages 

¢ Description: generates an XML file from a PF log 
which can then be optionally transformed into HTML 
or csv format 


PF writes its logs in a binary format, meaning that they 
cannot be read using head, tail, more, less, or an editor. 
While the logs can be read in real time using the com- 
mand tcpdump -n -e -ttt -i pflogo, It is sometimes con- 
venient to convert the logging information to another for- 
mat in order to study it and analyze trends. pflogx ren- 
ders the PF log in XML format and includes the ability to 
transform the XML into HTML or csv format. Optionally, 
the generated XML file can be passed to other third-par- 
ty tools for conversion to other formats. 

In order to use pflogx, the PF logging module must be 
loaded and at least one rule in the PF rulebase must in- 
clude the log keyword. You can double-check that log en- 
tries exist by typing pflogx -i /var/log/pflog. AS seen in 
this example, this command displays the log entries to the 
screen: Listing 2. 


Figure 3. Viewing Loaded Rules Using pftop 
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To instead save the log to an XML file, after the input 
(-i /var/log/pflog), specify the name of the output file 
(-o filename.xml). 

Optional filters can be placed between the input and out- 
put. They can be defined by action (-a pass Of -a drop), di- 
rection (-d in, -d out, Of -d in-out), protocol (-p icmp, -p 
ip, -p tcp, Of -p udp), and interface (-n interface name). 


If a filter is not included, all packets in the input log file 
will be generated to the output XML file. Several filter ex- 
amples can be found in the README file that is installed 
With pflogx. 

The package installs several XSLT files which are used 
to transform the XML file to HTML, XHTML, or csv format. 
To transform a generated XML file, copy it to the directory 


Listing 2. Sample XML File 


pflogx -i /var/log/pflog 

<?xml version="1.0" encoding="UTF-8”?> 

<pflogx version="0.86” > 

logs 

<log date="2013-04-23 12:43:48.261661” if="em0” action="drop” rule="2" direction="in” protocol="udp” src 

adt="205.233./5.201" “sre pporr— 123° “descvadz— T9268 I 7l “dese pore— 123°.) 

<log date="2013-04-23 12:44:24.41857” if="em0” action="drop” rule="2”" direction="in” protocol="(2)” src 

adr="192. 168 1.254" src port— “dest adr— 224.0.0.1" dest porit—"" /> 

<log date="2013-04-23 12:46:29.44070” if="em0” action="drop” rule="2”" direction="in” protocol="(2)” src 

adw=" 192,168. 2547 "sre pore "dest adi="224,0.0. 1" desu porr—" 7 7 2 

“log dare="2013-04-23-12°47;50. 295105" 11 em0 “action— drop” rule-"Z2Z" divection. in” protocol=] udp “src” 

adus V9Z716371 7) sre pore. 138° “desuvadn— 192 168.1 255 “desc pore— 133 7 - 

<log dace— 2013-04-23 12°47.50 296145" 11- em0™ action— drop xulle-"2° cirection— “in protocel— udp “sre, 
adr="192.168.1 71" sre port="138" dest adv—"192 168. 1.255" dest pore—" 138" /> 

(PeSteOr CURRIE Shipped...) 


| Date Interface (Action [Rule Direction (Protocol | Src. address Src. port |Dest. address [Dest. port 
[2013-04-23 12:43:48.261661 |emO drop [2 |in ludp (205.233.73.201|123 [192.168.1.71 123 
[2013-04-23 12:44:24.41857 |em0 drop [2 |in ((2) (192.168.1.254 | [224.0.0.1 | 
[2013-04-23 12:46:29.44070 [emo drop [2 iin ((2) (192.168.1.254 | [224.0.0.1 | 
[2013-04-23 12:47:50.298105 |emO drop [2 iin fudp (192.168.1.71 138 [192.168.1.255 ||138 
[2013-04-23 12:47:50.298145 lem0 drop |2 ~ |in ludp (192.168.1.71 138 [192.168.1.255 ||138 
[2013-04-23 12:48:34.46791 |em0 drop |2 iin ((2) (192.168.1.254 | [224.0.0.1 | 
[2013-04-23 12:48:52.827069 |em0 drop [2 |in ltcp (192.168.1.96 20261 192.168.1.71 ||22 
[2013-04-23 12:49:05.141466 |emO pass [5 fin licmp § |192.168.1.96 | [192.168.1.71 || 
2013-01-23 12:49:16.92832 |emO drop |2 jin  |tep -—«((i92.168.1.96 58203 [192.168.1.71 23 ~~ 
[2013-04-23 12:50:39.48931 |em0 drop [2 |in \(2) (192.168.1.254 | [224.0.0.1 | 
[2013-04-23 12:52:44.50937 |em0 drop [2 |in ((2) (192.168.1.254 | [224.0.0.1 | 
[2013-04-23 12:54:49.53355 |em0 drop |2 fin ((2) (192.168.1.254 | [224.0.0.1 | 
[2013-04-23 12:56:54.55460 |emo drop |2~ jin (|(2) |192.1681.254 | 224001 #=|| | 
[2013-04-23 12:58:59.57862 |em0 drop |2 iin ((2) (192.168.1.254 || [224.0.0.1 | 
2013-04-23 12:59:51.18030 jem0 drop [2 |in ludp (192.168.1.71 138 [192.168.1.255 138 
[2013-04-23 12:59:51.18055 |em0 drop [2 |in ludp (192.168.1.71 138 [192.168.1.255 138 
[2013-04-23 13:01:04.60149 |emO drop |2 (jin (2) 192.168.1.254 |  |224001 | © 
[2013-04-23 13:03:09.62345 |em0 drop [2 |in ((2) (192.168.1.254 | [224.0.0.1 | 
2013-04-23 13:05:14.64437 /em0 drop [2 |in ((2) (192.168.1.254 | [224.0.0.1 | 
[2013-04-23 13:07:19.67027 |em0 drop |2 ~~ |in ((2) (192.168.1.254 | [224.0.0.1 | 
[2013-04-23 13:09:24.69122 [emo drop [2 |in ((2) (192.168.1.254 | [224.0.0.1 | 
[2013-04-23 13:11:29.72991 |em0 drop [2 |in ((2) (192.168.1.254 | [224.0.0.1 | 
[2013-04-23 13:11:51.738657 |em0 drop |2 [in ludp (192.168.1.71 138 [192.168.1.255 |138 
[2013-04-23 13:11:51.738819 lem0 drop |2  |in ludp (192.168.1.71 138 1192.168.1.255 ||138 


Figure 4. Sample PF log in HTML Format 
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containing these files. On a FreeBSD or PC-BSD system, 
these files are located in /usr/local/share/examples/ 
pflogx/. In the generated XML file, the first line should be: 


<?xml version="1.0" encoding="UTF-8”?> 


Insert a second line that contains the name of the XSLT 
file. For example, to transform to HTML, add this line: 


<?xml-stylesheet type="text/xsl” href="export html.xs1”?> 


Save the edit and you should now be able to view the XML 
file in a web browser, as seen in the example in Figure 4. 


Listing 3. Sample pfstat Configuration 


# more /usr/local/etc/pfstat.conf 
collect 1 = interface “em0” pass bytes in ipv4 diff 
collect 2 = interface “em0” pass bytes out ipv4 
Guise 
image “/usr/home/dru/bandwidth.jpg” { 
from 7 days to now 
width 1000 height 400 


left 
Geapi lt bes. im bits 7e Icolon 0 3920 
filled 
16 Le lac 


checislan 2 sey Yorkie  Monhica ts vetollene 10). 0) Za 
} 


collect 3 = global states entries 
“/usr/home/dru/states.jpg” { 
from 12 months to now 

width 800 height 200 

left 


Graph 3 > Staces 


image 


WV \ 


enicries” Vcolor 200, 0 10 


pflogx provides a merge option (-m) which can be used 
to append new log entries to an existing XML file, allowing 
you to visualize the transformed log over time. 


pfstat 


¢ Website: http:/www.benzedrine.cx/pfstat.html 

¢ Availability: pkgsrc, FreeBSD and OpenBSD packages 

¢ Description: automatically generates graphs from PF 
statistics 


If you prefer to visualize the PF logs in a graph format, 
install pfstat. Once installed, create its log directory and 
log file if they do not exist: 


# mkdir /var/log/pflog 
# touch /var/log/pflog/pflog 


Next, create a configuration file named /usr/local/etc/ 
pfstat.conf. This file controls which graphs get creat- 
ed. A comprehensive file with comments on the various 
graphs it creates can be downloaded from http://www. 
benzedrine.cx/pfstat.conf. The following example shows 
a simpler configuration file which creates two graphs: 
one displays bandwidth in bits per second and the oth- 
er charts the state table. Edit the text in red to point to 
an existing directory path. The filename (e.g. bandwidth. 
jpg) should not already exist in the specified directory as 
pfstat will generate it for you. 

Next, type crontab -e as the superuser to edit the root 
user’s crontab. Add the following line: 


*/5 * * * * /usr/local/bin/pfstat -q >> /var/log/pfstat 


258.4 11.7 k 
232.3 k 10.5 k 
206.5 Kk 9.4 k 
180.7 k 8.2 k 
154.8 k 7.0 k 
hoi al 
§ 129.0 k 5.9k3 
r=) 3 
103.2 k 4.7 k 
77.4 K Jed K 
51.6 k 2.3 k 
25.8 k 1.2k 
-7 -6 4 “4 3 “2 “1 days 
in out 
Wed Apr 24 09:03:39 2013 
Figure 5. Sample Graph 
26 BSD 05/2013 


This instructs pfstat to query the logging interface ev- 
ery five minutes and to store the received logging infor- 
mation in its own database, which it uses to generate 
graphs. 

Finally, add this line to the beginning of /etc/pf£.con£ 
in order to set the logging interface. Replace emo with the 
name of the interface you wish to collect statistics on. Re- 
start the PF firewall after saving this edit. 


set loginterface em0 


Wait a bit (at least five minutes) to allow pfstat to add 
logging information to its database. The amount of infor- 
mation added to the database will depend upon how of- 
ten a logged rule matches the criteria you have config- 
ured pfstat to graph. 

Whenever you want to generate a graph, type pfstat 
-p. This instructs pfstat to read the entries in its data- 
base and to generate the images to the locations that you 
specified in /usr/local/etc/pfstat.conf. Figure 5 shows 
a sample /usr/home/dru/bandwidth. jpg from the configu- 
ration file above, after running pfstat for one day ona 
home desktop system. 

pfstat(8) provides some more information on how to use 
pfstat, remove old entries from the database, and query 
a remote host running pfstatd. 


pf£top, pflogx, and pfstat can be used to help the admin- 
istrator visualize the traffic flowing through a PF firewall. 
These utilities are easy to install and configure. If you are 
using the PF firewall, consider adding them to your admin- 
istrative toolkit. 
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FreeBSD Programming 
Primer - Part 4 


In the fourth part of our series on programming, we will 
continue to develop our CMS. Here we will examine how a 
modern CMS dynamically generates and controls content 
and implement a similar model in our PHP code. 


What you will learn... 
« How to configure a development environment and write HTML, 
CSS, PHP, and SQL code 


es were literally handcrafted masterpieces of content. 

Before applications such as Dreamweaver arrived that 
allowed content providers to design attractive pages with 
the ease of a document produced in a word processor, it 
was a matter of writing copious amounts of HTML for each 
page, checking that the links and the HTML were correct, 
and repeating for each page. This model was highly inef- 
ficient, as not only was a lot of the HTML repeated across 
pages, the chances of errors coming in and either caus- 
ing the page to render incorrectly or pointing to the wrong 
address became greater as the site grew. Managing a 
website with 100 pages is possible; a website with 10,000 
pages a nightmare. 

The complex sites we see today on the Internet would 
be impossible without the Content Management System. 
Yet even now, large innovative sites are moving away 
from the CMS model toward frameworks that consider the 
locally provided content to be only a part of the website 
with 3 party content supplying a significant proportion of 
the content. 

While the technology meets the ethos of the web in 
that data can be shared freely, it poses the web designer 
and brand manager with a huge challenge — how can 
we take disparate pieces of content and serve these in 
a “wrapper” that to our website visitors appears as if it 
seamlessly represents our brand values? How can we 
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What you should know... 


¢ BSD and general PC administration skills 


divorce the business process from the presentation? Is 
it possible for a website to develop a unique “personal- 
ity” while at the same time remaining fresh, dynamic and 
easily changeable? 

These hurdles are being overcome with the use of CSS 
(Cascading Style Sheets) and templating technologies. 
While the CSS manages the color, fonts, size, etc. of the 
content, templates allow us to adjust the order and vis- 
ibility of the content. For example, we want to generate 
widely different content (both from a stylized and literal 


Page ID: Content type: Rendering: style: 
- Reoord no 1 ' Page ' Field order Colour 
* News > Visible / hidden - Size 
* FAQ - Position 
DE |; 
= 
Page 1 
¥ Output 
http://mysite/page/1 


Figure 1. Page generation process 
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content perspective) depending on website section, page 
number and content type. See Figure 1 — Page genera- 
tion process. 


MySQL Interface 

As it is important that we can quickly test our CMS, for 
those that would prefer the “Cut, Paste and Click” ap- 
proach rather than managing long SQL statements via 
the command line, you can use a lightweight web-based 
database manager. The lightest of these (a single PHP 
page) is Adminer. An alternative is SQL buddy, and either 
of these can be quickly installed if desired by download- 
ing the archive and extracting into a folder under the /usr/ 
home/dev/data. The web-based interface can then be ac- 
cessed from: http://myserver/dirname. See Table 1 — Use- 
ful links. 


Adding New Content Types 

At the moment, we only have one content type — a page. 
This is stored in the pages table and holds the following 
content as shown in Table 1. 


Table 1. Page content from MySQL pages table 


id|title [ht [body 


This results in the following output as seen in Figure 2. 
Now let us create a second page in our database: 


Method 1 - Via CLI 


$ mysql -uroot -p’cms-password’; 


mysql> use freebsdcms; 


mysql> INSERT INTO ‘pages’ (‘title’, “hl°, ‘“body~) 


-> VALUES ‘HI’, 2°); 


(‘My second page’, 


4] PAGE HEADER 


Lorem Ipsum dolor sit amet, consectetur adipiscing elit. Mauris interdum auctor tellus sed dignissim. Phasellus non orci massa, nec feugiat sem. Vestibulum molestie interdum bibendum. 


Method 2 - Via saved SQL statement 
If you prefer, create a SQL file createpage2.sq/ in the SQL 
directory with the following content: 


USE freebsdcms; 
INSERT INTO ‘pages’ ( title’, ‘hl’, 


VALUES 


body’) 
(“My second page’, ‘HI’, *2’)>3 
Then execute this at the command line: 


S$ mysgl -uroot -p’cms-password’ < createpage2.sql 


Method 3 - Via Adminer / SQL Buddy 
Alternatively use the SQL command function in Adminer 
to execute the following SQL statement: 

INSERT INTO ‘pages’ (‘title’, ‘hl’, 
VALUES 


body") 
(‘My second page’, ‘Hl’, ‘2'); 

Houston, We Have a Problem 

We now have two pages in our database, but index.php 
still contains the following code: 


// Build page - use first record in database 


Spage[‘id’] = 1; 
buildpage (Spage) ; 


This hard-wires index.php to only serve a page with an 
ID of 1. Depending on the URL passed to the webserver, 
we want to serve that type of content. For example http:/ 
mysite/pages/1 will serve a page with an ID of 1, where- 
as http://mysite/faqs/1 will serve an FAQ with an ID of 1, 
etc. Visiting http://mysite will return the home page (Page 
1). This leads us to the next problem — where do we 
store the content types? We could include this in a sep- 
arate MySQL table, but this would require an addition- 
al SQL query to be executed every time a page is load- 
ed. As content types will not be changed very often, we 
can create another include file that defines our content 


Nune quis elit nulla, sit arnet rutrum lorem. Quisque odio est, sagittis nec accumsan ut, placerat sit amet lectus. Curabitur aliquam dignissim felis, a malesuada leo fringilla at. Sed ornare 


aliquet lacus, quis impérdiet augue mattis ¢u. Nulla porta odio ut erat consectetur at molestie justo suscipit. Aenean convallis pellentesque nisl, vitae posuere mauris facilisis vitae. Morbi 


In tellus nisl, vel facilisis diam. 


Figure 2. Our first page 
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Listing 1. content.inc 


<?php 
x 
* 
A <COneenia uC 
* Defines content types for our CMS 
* 
7 
// Define the content type. This must match any tables 


defined in our 


ye SCuls 

PCONMECHIEREYpes "= page’; 
SCOMESMIE TOSS || = weKel s 
PCOMECHIE NEYPesile— Mews: 


// Map each content type to a table. Each content type 
must be matched 


// to a corresponding table 


SCromesine ielolSes || Wee’ || =" isccies’ s 
veommemic ieclolkes | Viger’ | > ieecieps” 2 
PcOllen en eables | hems | = ‘news’; 


Listing 2. pages_template.inc 
<7 oe 
/ ae 

k 

Pages e tema tee 


* Template for our page content type 


* 


* For content type foo the corresponding template would be: 


E eO Ome modecive slnie 


a To disp bay a cie ld : 


eo Vigeinleleia (Sieigevile:|| ieicimles ua suledLel, cys Voleumayerel iia ele” ||) hr 


* To hide a field omit it from here 


* To change the rendering order, just re-order the fields 


* NOTE: Any content generated by javascript will not be 
managed here 
e A CLOSING, => tag) Vs mandarory 
x 
/ 
render (Stheme[‘title’]); 
render (Stheme[ ‘debug’ ]) ; 
render (Stheme[‘h1’]); 
render (Stheme[ ‘timestamp’ ]); 
render (Stheme[ ‘body’ ]); 
render (Stheme[ ‘licence’ ]); 


Ca 


Listing 3. index.php replacement code 

// First we need to parse the URL that was passed to us 
to extract the 

// id and the content type. 

SURI = Pook VEE | RROURST URI 3); 

if (SURI == ‘/'){ 


// If this is a request to root (/) redirect to page 1 


Srequest = array(‘pages’,1); 


buildpage (Srequest) ; 


jelse{ 


// Parse the request, if it is valid get the content 


trom Ou DE 


prcgUes@essPabceeroqucsu( Uni), 


aie (Dake mG (Sretsepitersic)) 4 


buildpage (Srequest) ; 


jelse{ 


echo “Invalid request”; 


Listing 4. core.inc replacement code 


function buildpage(Srequest) { 


/ / Content. catini elon. 


require INCLUDES.’ content.inc’; 


// Routes our incoming request to the right content 
type and pulls 
// she content from out Ds. 


PeOneenuleype — »Lequese| Ul), 
Sid = Srequest[1]; 
PEChplabe les — MEANT Se COnmehiE Oc us 


template.inc’; 


// Build the SQL and get the result 
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sql = “SELECT * FROM Scontent type WHERE id=’Sid’ LIMIT 1”; 
Sicesuillic = mhyeell eeilece (ciel) 


// Check we have some content to display 


hie ((Siassollae [0 Pe—— (0 


echo ‘No data’; 


return; 


// Check we have a template file 


ci iilevextsirs( tenmelace ile) ja 


echo ‘No template’; 


return; 


} 


// Don't write any output to browser yet as we want 
(EO) JOSIE sO s, 

// our content using a theme. If enabled use our 
Ope AMZ ak kon 


// callback to remove white space etc. 
Obwstare( Opuimizeucaliback )7; 

// Output our page header 

outfile (TEMPLATES ‘header.inc’); 

/7 Creace our body 

echo wraptag(‘title’, Sresult[‘title’]); 
echo HEAD; 


echo BODY; 


// Generate a unique ID based on content type 


// Map the requested content type from our real table name 


SCE = chee Sechieeln(Sccmeeme eye, Veemesme jeelolliss)) 7 


SchOs “<“cliiy sid="" Joces, >; 


// If we are in debug mode, show an alert 


if (DEBUG) { 


Stheme[‘debug’] = div(‘&para;’, ‘’, ‘debug’); 


// Dump the title & id out to our theme template 
Stheme | ‘id’ | = Sresult| “id? |; 
Stheme| ‘titie’ | = Sresult| ‘title’ |; 
// As we don’t know how many fields we will have in 
OUT sCONeene 
// type after our id, iterate through each in turn and wrap 


oy) Se Slats MatkadMo stab elemecmecaiy: 


Soffset = Sresult[l) = 1; 
Spos = 0; 


foreach(Sresult as Skey => Svalue) { 


if(Spos > Soffset) { 


Stheme [Skey] = div(Sresult [Skey], Skey.’-’.Sid, Skey); 


SOO sear e 


// Add our standard copyright notice 


Stheme[‘licence’] = div(ahref (COPYRIGHT, LICENCE, ‘Copyright and 


lacence detaiis’) "7 bircence’ ); 


// Include our template file 


igSC Mice Ones (SciSile lens AS) 3 


// Close our content type tag 


echo *</div >> 


// Output our HTML page footer 
outfile (TEMPLATES ‘ROCHE. 1mc’ ) 


/jeP lush tt all ous and drsp lay 


ob end Hush {); 
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Listing 5. core.inc additional code 


function [SeneeS siclepbles ic SWIRL) 


// Returns the type of content and the ID 
// of the content requested. 

// parse request (/page/1) 

Vy eo aieiees || eage. 4h) 

// parse request (/rubbish/123456) 

// NULL 


Jj) Content cetinie1ons 


Geguive once INCMUDES = cOnrenicn mie | 


syvere = INN 
Sale, = INU 
Siwenlane, = 10% 


// Fetch the parameters from the URL 


Sarray = explode(‘/’,SURI) ; 


if Werdon’ teneco the first */ =-celete Ene first 


empty 


// array item 


eel = eucicely sldulirie (Giclee): 


// Check we have 2 parameters 


Sparamcount = count (Sarray) ; 


if (Sparamcount == 2) { 


// First test passed - We have 2 parameters 


Svalid ++; 


BOP 

Q 

ii 
i 


Sams G@acyanlle 
Sid 


patie val lle 


ti inparhea (oCu, eCOnuenu sey pes) ): 


// Tf content type matches our list second test 


passed 


Svalid ++; 


// Map the requested content type to our real 


table name 


peuccely (0) = Seemnsimc iceloles | oer]: 


alan (ales) oubbnyes @aliol ((yakel)) | 


// If ID is a number, third test passed 


Svalid ++; 


Toe (leayel lake ==" sy) 


// Valid parameters passed, return content type 


and page ID 


return Sarray; 


jelse{ 


jj les tanked = retired INDI 


retura NULL: 


PUNCe VON OpEIMiuze cea Liback( (burner Fr 


// Replace all spaces and cruft between tags 


at (OR MiMINZE 


ebu= Pred replace ( => \st<~" 7) ><) 7 Sburter):, 
S10 =" jorge) isSiOllerers (ie Val ie | aly Sis) § 
Sb = preg replace(‘!\st!’, ‘ *, $b); 


Pete uicm polos 
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Language: | English ¥ MySOL » localhost » freebodems » SQL command 


Adminer 3.6.4 SOL command 


SOL command Dump Logout 


IMSERT INTO ‘pages ([ title, hil, 
WALUES (‘My second page’, ‘H1", ‘2° 


frdustepehorrs LJ 


Create new table 
select pages 


File upload: Choo: File | Mio fil cheer (< 2MB) 


Show only errors 


Exnoute Stop on error 


From server) j~ History 


Figure 3. Using Adminer to execute SQL statement 


Listing 6. mysql.inc replacement code 
SHjolme 
/* 
x 
Fins Glee alee 
HWeoncaiis, My SOla tiie huOMs sh olewouianCM > 
* 
as 
EUMeE LOM Mysql select scl) ae 


116 |Sielo:— ere dialSele, euciciavey 2 (0) | 
die(‘Unable to connect to database ['‘ 


Eee 


pab= COMMccENCELOn 


if ('Sresult = $db->query ($sql) ) { 
if (DEBUG) { 
die(‘There was an error running the query 


[ =Sdb-Serror.” |7)> 


// Pass our results to an array to be returned 
Sr = array(); 


Sr[] = S$result->num_rows; // No of rows returned 


Sdb = new mysqli(DBSERVER, DBUSER, DBPASSWORD, CMSDB) ; 


(7) Nomor scolummas tm calle 


Sa (| 
ce (i 


eycllo)—Panledkiel tereihane 5 
pob>-abbecved 1ows, 
update / delete 


// NOwOE COWS abreerce e 


// Append the results to our result count 


te(eresule=-num cows 1 = 0) 


Cig = cudiechy ils (Sie, Sisesulle rete suse (UN ONE 
INSISIOIC 


J SESS tele ae Seie 


Sresult->free(); 


J] VCloce the connection 


Sdb->close(); 


asieilain Siac 


} 
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Your donations have helped make FreeBSD 
the best OS available! By investing in 

the services provided by The FreeBSD 
Foundation you have helped us fund projects 
to keep FreeBSD a high-performance, 
secure, and stable OS. 


What will the Foundation accomplish with your 
donation in 2013? 


e Software development projects for FreeBSD: 
$600,000. 


e Paid staff time supporting Release 
Engineering and Security teams. 


¢ Grow staff: Five technical staff members by 
year-end. 


e Provide support for BSD conferences 
around the globe, in Europe, Japan, Canada, 
and the USA. 


e Hardware to maintain and improve FreeBSD 
project infrastructure: $130,000. 


¢ FreeBSD community growth through 
marketing and outreach to users and 
businesses. 


e Legal services and counsel protecting the 
FreeBSD trademarks. 


| | j 


| 


“Support 


FreeBsD 
by donating 


FreeBSD is internationally recognized as an innovative 
leader in providing a high-performance, secure, and stable 
operating system. Our mission is to continue and increase 
our support and funding to keep FreeBSD at the forefront of 
operating system technology. But, we can’t do this without 
your help! 


Last year with your generosity, we raised over $770,000. This year we will invest $1,000,000 
to support and promote FreeBSD. 


We have kicked off the new year with three newly funded projects, and are actively 
soliciting additional project proposals. 


Please support the Foundation during our Spring Fundraising Drive, and help us raise 
$100,000 from 1000 donors between April 15th and May 30th. 


we need your help. 
we can't Ao this without you... 


Make your donation today. Go to: 
www.freebsdfoundation.org/donate 


) Th Q Then talk to your employer 
about matching your gift— or 
{ eC eC making their own donation. 


FOUNDATION 


Find out more, visit: 


AaAfreebsdfoundation.org 
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types. We can then automatically use a custom template 
depending on the content type to post process our spe- 
cific content. 

First of all, we need to make some modifications to 
Apache so that it serves our index.php page as default. 
Edit the line in /usr/local/etc/apache22 /httpd.conf to 
match the following: 


DirectoryIndex index.php 


Find the section marked <Directory ‘/usr/local/www/ 


apache22/data”> and add the following: 


# 
# Redirect on error via our CMS 


it 


ErrorDocument 401 /index.php 
ErrorDocument 403 /index.php 


Listing 7. html.inc replacement code 


<?php 
ies 
* 


~ Wen eta 


‘WC OMe sie Oce ihn arin ho mise not aOulawe MS 
Ze 
“gy 


function wraptag(Stag, Stext) { 


// Wraps Stext with compliant tags 


// wraptag(‘p’,sometext) 


// <p>sometext</p> 


PS wUae  <G'e aC ia ee IEeK EE a ye ee oieag. 9 a es 


EUNCEION divisdiveontent, celass,, Sid =) 7). 4 


// Generates a div tag Stext with compliant tags 


jy Oder MCOniemte. foc lass.) 

// <div class="class”>content</div> 

jy)  Oier( “COnueiiL class 7 20”) 

// <div id="id” class="class”>content</div> 
1 clara Gola) aherssanevers sam cane liad 


i atin a= "Ge Comeene dint 


ji Gear “Gone as! os.) ) 


7) <Gave content.) div> 


cl (CS) alo Bae fae 


Biel =) ide. Giicl 


fie oelass. “(= ) a4 


HALAL 


Solace — 2 (cilacc— 2 S@llage M7: 


‘ein buce <““eunys ~ 8 Save, 9 Silica 24S 


Sch veleimesiie_ | </ellyes 


FUNCT VOM anrer(Stext, suri, Stitie = %")) 4 


// Generates an href tag $text with compliant tags 


// ahref (‘Click here’, freebsd.org) 
// <a href="http://freebsd.org” title="Click 


here =C biocwnere—/ a— 


// anref(‘Click here’, freebsd.org,’ Link title’) 


// <a href="http://freebsd.org” title="Link 


Perle Cliek nere-7 a 


if ($title == ‘’) { 


Siegbeids = Sonell 


Select ——. “ouch — 6 emo Ural ne) enedile—\ 7 a tro baktelbe 


Mi ep olcemieases jar. 


feaieiikeig Syellaieteuz ¢ 


function render (Sfield) { 


// Renders via template 


Schou site le. 
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ErrorDocument 404 /index.php 
ErrorDocument 500 /index.php 


This will force all traffic to be passed to our index.php for 
processing. As root, delete our unwanted files then re- 
start Apache: 


S rm /home/dev/data/index.xhtml 
S rm /home/dev/data/index.html 


S apachectl restart 


When you visit http://mysite or http://mysite/, page 1 
should be displayed. Now for the modifications that will 
facilitate content type routing and theme control. Create 
a file in the includes directory called content.inc with the 
content from Listing 1. 

Create the following template file pages_template.inc in 
the templates directory shown in Listing 2. 

Remove the following section entirely from index.php: 


// Build page - use first record in database 


Spage[‘id’] = 1; 
buildpage (Spage) ; 


Replace with the one shown in Listing 3. Remove entire- 
ly the function call buildpage(Spage) from core.inc. Re- 
place with the code shown in Listing 4. Add the function 
calls from Listing 5 to the end of core.inc. 

Replace html.inc with Listing 7. Append the following to 
cms.inc: 


// Optimize output by removing white space between tags etc. 


define (“OPTIMIZE”, true); 


Errata 
In the previous article of this series the following syntax was 
incorrect: 


#dev mysgl -u root password ‘cms-password’ < 
createdb.sgql 
#dev mysgl -u root password ‘cms-password’ < createpagetbl.sql 


#dev mysql -u root password ‘cms-password’ < createpage.sql 
It should have read: 


#dev mysgl -u root -p’cms-password’ < createdb.sql 
#dev mysgl -u root -p’cms-password’ < createpagetbl.sql 


fdev mysql =u root —p’ cms—-password’ < createpage. sql 


Our apologies. 
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Useful Links 
SQL buddy -— http:/sqlbuddy.com 
Adminer — http://www.adminer.org 


Testing and Adding New Content 
That is a lot of code we have added, but we now have a 
major jump in functionality. We can create any number of 
content types now by creating a new table (e.g. fag, news, 
etc.) The only essential fields we must define are ID and 
TITLE. After these two fields you may define as many or 
as few as you require. You will need to create a match- 
ing template file with the fields you want to display or else 
the content will be unable to render. Once you have add- 
ed new records to your content type (Adminer makes this 
quick and easy), the content can be accessed via your 
browser at: http://mysite/mycontenttype/mypageid. |f you 
attempt to access invalid content, you will be presented 
with a rudimentary error message. 

In the next article in the series, we will look at theming in 
detail and how we can lay out the site using a combination 
of templates and CSS. 


ROB SOMERVILLE 

Rob Somerville has been passionate about technology since his 
early teens. A keen advocate of open systems since the mid-eight- 
ies, he has worked in many corporate sectors including finance, 
automotive, airlines, government and media in a variety of roles 
from technical support, system administrator, developer, systems 
integrator and IT manager. He has moved on from CP/M and nixie 
tubes but keeps a soldering iron handy just in case. 
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DTrace 


A Deeper Approach 


In my article “Intro to DTrace’, published in May 2012 in BSD 
Magazine, | described DTrace all the way from configuring 
your system to enabling DTrace probes to executing some 
D scripts in order to show you some DTrace features. This 
article will take a deeper approach to DTrace. 


place in the DTrace kernel module. Each probe 
definition is composed of the four elements sepa- 
rated by colons. The general form is: 


yT he processing and buffering of all probe data takes 


provider:module: function:name 


Provider 
A provider is a DTrace kernel module, which logically 
groups together various probes that are related. Exam- 


Table 1. D Macro Variables 


S[0-9]+ |= macro arguments look at macros 
Segid effective group-ID getegid(2) 
Seuid effective user-ID geteuid(2) 
Sgid real group-ID getgid(2) 

Spid process ID getpid(2) 
Spgid process group ID getpgid(2) 
Sppid parent process ID getppid(2) 
Sprojid project ID getprojid(2) 
Ssid session ID getsid(2) 
Starget target process ID see target process id 
Staskid task ID gettaskid(2) 
Suid real user-ID getuid(2) 
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ples of providers in DTrace include: fbt, which instruments 
kernel functions; pid, which instruments userland pro- 
cesses; and syscall which instruments system calls. 


Module 

A module is the program location of the group of probes. 
This could be the name of a kernel module where the 
probes exist, or it could be a userland library. Example 
modules are the libc.so library or the ufs kernel module. 


Function 

Specifies the specific function which this probe should fire 
on. This could be something like a particular function in a 
library SuCN aS print£() Of strcepy(). 


Name 

This is usually the meaning of the probe. Sample names 
are “entry” or “return” for a function or “start” for an I/O 
probe. For instruction level tracing, this field specifies the 
offset within the function. Understanding this allows you to 
understand the purpose of a particular probe. You can list 
all the probes on a DTrace instrumented system by provid- 
er by running the dtrace -1 command. It will list the probes 
in the format described above. If one of them is missing, it 
will be taken as a wildcard. It could be written as: 


provider::function:name or provider:*:function:name 
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NET SERVICES IS AN APPLICATION HOSTING COMPANY FOCUSED 


ON OPEN SOURCE APPLICATIONS MANAGEMENT IN HIGH AVAILABILITY ENVIRONMENT. 


NET OPEN SERVICES IS PROUD TO PROVIDE A HIGH QUALITY SERVICE TO OUR CUSTOMERS SINCE 10 YEARS. 
OUR EXPERTISE INCLUDES: 


(OPENSTACK, CLOUDSTACK, RED HAT ENTERPRISE VIRTUALIZATION) 
REMOTE MONITORING AND MANAGEMENT 24/7 


NETWORKING AND SECURITY = 
(OPEN BSD, IP TABLE, CHECKPOINT, CISCO...) 


OS AND APPLICATION MANAGEMENT 
(FREE BSD, OPEN BSD, SOLARIS, UNIX, LINUX, AIX, MS WINDOWS) 


DATABASE MANAGEMENT _ ron ante 
(ORACLE, MYSQL, CASSANDRA, NOSQL, MS SQL, SYBASE...) 


MANAGED HOSTING IN CARRIER CLASS DATA CENTERS 
DISASTER RECOVERY 


CLOUD COMPUTING, PUBLIC, PRIVATE AND HYBRID CLOUD MANAGEMENT | 


Net 


SERVICES 


WE PROVIDE SERVICES IN EVERY STEP OF THE PROJECT LIFE, DESIGN, DEPLOYMENT, MANAGEMENT AND EVOLUTIONS. 
NETOPENSERVICES TEAM INCLUDES EXPERIENCED LEADERS AND ENGINEERS IN THE INTERNET SERVER INDUSTRY. 


QUR TEAM HAS 15 YEARS OF EXPERIENCE IN DEVELOPING INTERNET INFRASTRUCGTURE-GRADE SOLUTIONS AND PROVISIONING INTERNET 
DATACENTERS AND GLOBAL SERVICE NETWORKS TOGETHER. 


WE OFFER EXCEPTIONAL HARDWARE SUPPORT AS SOFTWARE SUPPORT ON UNIX/LINUX AND OPEN SOURCE APPLICATION. 


NETOPENSERVICES DELIVERS THESE CUSTOM-BUILT LINUX AND UNIX SERVERS, AS WELL AS PRECONFIGURED SERVERS AND SCALABLE STORAGE 
SOLUTIONS, TO OUR CUSTOMERS. WE ALSO OFFER CUSTOM DEVELOPMENT AND ADVANCED-LEVEL UNIX/LINUX CONSULTING SOLUTIONS. 


WWW.NETOPENSERVICES.COM ¢ CONTACT@NETOPENSERVICES.COM 
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Table 2. DTrace Built-in Variables 


int64_t arg, ..., arg9 


args[] 


uintptr_t caller 
chipid_t chip 
processorid_t cpu 
cpuinfo_t *curcpu 


lwpsinfo_t *curlwpsinfo 
psinfo_t *curpsinfo 


kthread_t *curthread 


string cwd 


uint_t epid 


int errno 
string execname 
gid_t gid 
uint_tid 


uint_t ipl 


Igrp_id_tlgrp 
pid_t pid 

pid_t ppid 

string probefunc 
string probemod 
string probename 
string probeprov 
psetid_t pset 
string root 

uint_t stackdepth 
id_ttid 


uint64_t timestamp 


uid_tuid 
uint64_t uregs[] 
uint64_t vmregs[] 


uint64_t vtimestamp 


uint64_t walltimestamp 
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The first ten input arguments to a probe represented as raw 64-bit integers. If fewer than ten arguments are 
passed to the current probe, the remaining variables return zero. 


The typed arguments to the current probe, if any. The args[] array is accessed using an integer index, but each 
element is defined to be the type corresponding to the given probe argument. For example, if args[] is referenced by 
a read(2) system call probe, args[0] is of type int,args[1] is of type void *, and args[2] is of type size_t. 


The program counter location of the current thread just before entering the current probe. 
The CPU chip identifier for the current physical chip. 

The CPU identifier for the current CPU. 

The CPU information for the current CPU. 


The lightweight process (LWP) state of the LWP associated with the current thread. This structure is described 
in further detail in the proc(4) man page. 


The process state of the process associated with the current thread. This structure is described in further 
detail in the proc(4) man page. 


The address of the operating system kernel’s internal data structure for the current thread, the kthread_t. 
The kthread_t is defined in<sys/thread.h>. Refer to Solaris Internals for more information on this variable and 
other operating system data structures. 


The name of the current working directory of the process associated with the current thread. 


The enabled probe ID (EPID) for the current probe. This integer uniquely identifies a particular probe that is 
enabled with a specific predicate and set of actions. 


The error value returned by the last system call executed by this thread. 
The name that was passed to exec(2) to execute the current process. 
The real group ID of the current process. 


The probe ID for the current probe. This ID is the system-wide unique identifier for the probe as published by 
DTrace and listed in the output of dtrace -l. 


The interrupt priority level (IPL) on the current CPU at probe firing time. Refer to Solaris Internals for more 
information on interrupt levels and interrupt handling in the illumos operating system kernel. 


The latency group ID for the latency group of which the current CPU is a member. 
The process ID of the current process. 

The parent process ID of the current process. 

The function name portion of the current probe’s description. 

The module name portion of the current probe's description. 

The name portion of the current probe’s description. 

The provider name portion of the current probe's description. 

The processor set ID for the processor set containing the current CPU. 

The name of the root directory of the process associated with the current thread. 
The current thread’s stack frame depth at probe firing time. 


The thread ID of the current thread. For threads associated with user processes, this value is equal to the 
result of a call to pthread_self(3C). 


The current value of a nanosecond timestamp counter. This counter increments from an arbitrary point in the 
past and should only be used for relative computations. 


The real user ID of the current process. 
The current thread's saved user-mode register values at probe firing time. Use of the uregs[] array is discussed in 
The current thread’s active virtual machine register values at probe firing time. 


The current value of a nanosecond timestamp counter that is virtualized to the amount of time that the 
current thread has been running on a CPU, minus the time spent in DTrace predicates and actions. This counter 
increments from an arbitrary point in the past and should only be used for relative time computations. 


The current number of nanoseconds since 00:00 Universal Coordinated Time, January 1, 1970. 
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Macro Variables 

The D compiler defines a set of built-in macro variables 
that you can use when writing D programs or interpreter 
files. Macro variables are identifiers that are prefixed with 
a dollar sign ($) and are expanded once by the D compiler 
when processing your input file. The D compiler provides 
the following macro variables, shown in Table 1. 


Built-in Variables 

Table 2 provides a complete list of D built-in variables. All 
of these variables are scalar global variables; no thread- 
local or clause-local variables or built-in associative ar- 
rays are currently defined by D. 


Macro Arguments 

The D compiler also provides a set of macro variables cor- 
responding to any additional argument operands speci- 
fied as part of the dtrace command invocation. These 
macro arguments are accessed using the built-in names 
$0 for name of the D program file or dtrace command, $1 
for the first additional operand, $2 for the second operand, 
and so on. If you use the dtrace -s option, $0 expands to 
the value of the name of the input file used with this op- 
tion. For D programs specified on the command-line, $0 
expands to the value of argv[O] used to exec DTrace itself. 

For example: 


#!/usr/sbin/dtrace -s 


syscall::write:entry 
/pid == $1/ 

{ 

} 


Target Process ID 

Use the starget macro variable to create scripts that can 
be applied to a particular user process of interest that is 
selected on the DI race command line using the -p option 
or created using the -c option. The D programs specified 
on the command line or using the -s option are compiled 
after processes are created or grabbed and the starget 
variable expands to the integer process-ID of the first 
such process. For example, the following D script could 
be used to determine the distribution of system calls ex- 
ecuted by a particular subject process: 


syscall:::entry 
/pid == Starget/ 
{ 


@[probefunc] = count(); 
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Subroutines 

Subroutines differ from actions because they generally 
only affect internal DTrace state. Therefore, there are no 
destructive subroutines, and subroutines never trace data 
into buffers. Many subroutines have analogs in the Sec- 
tion 9F or Section 3C interfaces... 


alloca 
wold, “alloca (size t. S176) 


alloca allocates size bytes out of scratch space, and re- 
turns a pointer to the allocated memory. The returned 
pointer is guaranteed to have 8—byte alignment. Scratch 
space is only valid for the duration of a clause. Memory 
allocated with alloca will be deallocated when the clause 
completes. If insufficient scratch space is available, no 
memory is allocated and an error is generated. 


basename 
string basename(char *str) 


basename is a D analogue for basename(1). This sub- 
routine creates a string that consists of a copy of the 
specified string, but without any prefix that ends in /. The 
returned string is allocated out of scratch memory, and 
is therefore valid only for the duration of the clause. If in- 
sufficient scratch space is available, basename does not 
execute and an error is generated. 


bcopy 
vod -bcopy (void. “src, void. “dest; Suze t. size) 


bcopy copies size bytes from the memory pointed to by src 
to the memory pointed to by dest. All of the source memo- 
ry must lie outside of scratch memory and all of the desti- 
nation memory must lie within it. If these conditions are not 
met, no copying takes place and an error is generated. 


cleanpath 
string cleanpath(char *str) 


cleanpath creates a string that consists of a copy of the 
path indicated by str, but with certain redundant ele- 
ments eliminated. In particular /./ elements in the path 
are removed, and /../ elements are collapsed. The col- 
lapsing of /./ elements in the path occurs without regard 
to symbolic links. Therefore, it is possible that cleanpath- 
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could take a valid path and return a shorter, invalid one. 
For example, if str was /foo/../bar and /foo was a sym- 
bolic link to /net/foo/export, Cleanpath would return the 
string /bar even though bar might only be in /net/foo 
not/. This limitation is due to the fact that cleanpath is 
called in the context of a firing probe, where full symbol- 
ic link resolution or arbitrary names is not possible. The 
returned string is allocated out of scratch memory, and is 
therefore valid only for the duration of the clause. If insuf- 
ficient scratch space is available, cleanpath does not ex- 
ecute and an error is generated. 


copyin 
void *Copyin(uintptr t addr, size t 217¢@) 


copyin copies the specified size in bytes from the spec- 
ified user address into a DTrace scratch buffer and re- 
turns the address of this buffer. The user address is in- 
terpreted as an address in the space of the process as- 
sociated with the current thread. The resulting buffer 
pointer is guaranteed to have 8-byte alignment. The ad- 
dress in question must correspond to a faulted-in page 
in the current process. If the address does not cor- 
respond to a faulted-in page, or if insufficient scratch 
space is available, NULL is returned, and an error is 
generated. See Chapter 33, User Process Tracing for 
techniques to reduce the likelihood of copyin errors. 


copyinstr 
string copyinstr(uintptr t addr) 


copyinstr copies a null-terminated C string from the 
specified user address into a DTrace scratch buffer and 
returns the address of this buffer. The user address is in- 
terpreted as an address in the space of the process as- 
sociated with the current thread. The string length is lim- 
ited to the value set by the strsize option. As with copy- 
in, the specified address must correspond to a faulted- 
in page in the current process. If the address does not 
correspond to a faulted-in page, or if insufficient scratch 
space is available, NULL is returned and an error is gen- 
erated. 


copyinto 
vyold, COpyinto (uintptr vt. addr, S176 t size, vold *dest) 


copyinto copies the specified size in bytes from the 
specified user address into the DTrace scratch buffer 
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specified by dest. The user address is interpreted as an 
address in the space of the process associated with the 
current thread. The address in question must correspond 
to a faulted-in page in the current process. If the address 
does not correspond to a faulted-in page, or if any of the 
destination memory lies outside scratch space, no copy- 
ing takes place and an error is generated. 


dirname 
string dirname(char *str) 


dirname is a D analogue for dirname(1). This subroutine 
creates a string that consists of all but the last level of the 
pathname specified by str. The returned string is allocat- 
ed out of scratch memory, and is therefore valid only for the 
duration of the clause. If insufficient scratch space is avail- 
able, dirname does not execute and an error is generated. 


Iltostr 


string lltostr(long long num) 


string lltostr(long long num, int base) 

lltostr is a D analogue for strto11(). This subroutine cre- 
ates a string that represents the value of nun. If base is 
specified, then num Is interpreted in that base. 

msgdsize 

size t msgdsize(mblk t *mp) 

msgdsize returns the number of bytes in the data message 
pointed to by mp. See msgdsize(9F) for details. msgdsize only 
includes data blocks of type m _ pata in the count. 

msgsize 

size t msgsize(mblk t. *mp) 

msgsize returns the number of bytes in the message 
pointed to by mp. Unlike msgdsize, which returns only the 
number of data bytes, msgsize returns the total number 


of bytes in the message. 


mutex_owned 
int mutex owned(kmutex t *mutex) 


mutex owned IS an implementation of mutex _ owned(9F). 
mutex owned returns non-zero if the calling thread cur- 
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rently holds the specified kernel mutex or zero if the 
specified adaptive mutex is currently unowned. 


ia 
mMmilirayvy nwunar 
mutex ownel 


kthread t *mutex owner (kmutex t *mutex) 


mutex owner returns the thread pointer of the current 
owner of the specified adaptive kernel mutex. mutex _ 
owner returns NULL if the specified adaptive mutex is 
currently unowned or if the specified mutex is a spin mu- 
tex. See mutex _ owned (9F). 
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int mutex type adaptive (kmutex t *mutex) 


mutex type adaptive returns non-zero if the specified 
kernel mutex is of type muTEx ADAPTIVE, Or zero if it is 
not. Mutexes are adaptive if they meet one or more of 
the following conditions: 


¢ The mutex is declared statically 

¢ The mutex is created with an interrupt block cookie of NULL 

e The mutex is created with an interrupt block cookie 
that does not correspond to a high-level interrupt 


See mutex init(9F) for more details on mutexes. The 
majority of mutexes in the illumos kernel are adaptive. 


int progenyof (pid _ t pid) 


progenyof returns non-zero if the calling process (the 
process associated with the thread that is currently trig- 


Table 3. SPARC uregs[] Constants 


(NS aa eS NE D etary 
| Constant | R SLT 


uA 


R_GO..R_G7  %g0..%g7 global registers 
R_O0..R_O7 %00..%07 out registers 
R_LO..R_L7 %10..%I7 local registers 
R_10..R_I7 %i0..%i7 in registers 


R_CCR Y%ccr condition code register 

R_PC Y%pc program counter 

R_NPC Ynpc next program counter 

R_Y Y%y multiply/divide register 

R_ASI Y%asi address space identifier register 
R_FPRS %fprs floating-point registers state 
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GEEKED AT BIRTH 


You can talk the talk. 
Can you walk the walk? 


IT’S IN YOUR DNA 


TIPS & TRICKS 


gering the matched probe) is among the progeny of the 
specified process ID. 


rand 
int rand(void) 


rand returns a pseudo-random integer. The number re- 
turned is a weak pseudo-random number and should not 
be used for any cryptographic application. 


rw_iswriter 
int rw_iswriter(krwlock t *rwlock) 


rw iswriter returns non-zero if the specified read- 
er-writer lock is either held or desired by a writer. If the 
lock is held only by readers and no writer is blocked or if 
the lock is not held at all, rw iswriter returns zero. See 


Ww. Inte (SF x 

rw_write_held 

int rw_write held(krwlock_t *rwlock) 

rw write held returns non-zero if the specified read- 
er-writer lock is currently held by a writer. If the lock is 
held only by readers or not held at all, rw write _ 


heldreturns zero. See rw _ Ate Or) 


speculation 

int speculation (void) 

speculation reserves a speculative trace buffer for 
use with speculate and returns an identifier for this 
buffer. 

strjoin 

SLYring Strjoin (Ghar “strl, char *strZ) 

strjoin Creates a string that consists of str1 concate- 
nated with str2. The returned string is allocated out of 
scratch memory and is therefore valid only for the dura- 
tion of the clause. If insufficient scratch space is avail- 


able, strjoin does not execute and an error is generated. 


strlen 


Size... ecrlen{string Scr) 
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Table 4. x86 uregs[] Constants 


R_GS %gs 


R_DS %ds 


R_ESI %esi 


R_EAX %eAaX 


R_EAX %eAaX 


R_ECX %ECX 


R_TRAPNO %trapno 


R_EIP %eip 


R_ERR %err 


R_UESP Yuesp 


Table 5. amd64 uregs[] Constants 


R_RFL %rfl 


R_RAX %rax 


R_RDX %rdx 


R_RBP %rbp 


R_RDI Y%rdi 


%r9 


%r11 


%r13 


%r15 
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strlen returns the length of the specified string in bytes, 
excluding the terminating null byte. 


tolower 
String (char *str) 


tolower returns a new string which is the lowercase ver- 
sion of str. 


toupper 
String (char *str) 


toupper returns a new string which is the uppercase ver- 
sion of str. 


Creating Debugging Tools 

First Case Scenario 

Let's suppose we have an application that segfaults when 
trying to execute and instruction at address 040404040, 
this is clearly an overflow. With DTrace, we can stop the 
program before it crashes trying to execute the instruction 
at this address. This allows us to carry out data collection 
and analysis, such as printing CPU register values, func- 
tion parameters, dumping memory: 


#/usr/sbin/dtrace -s 
pidStarget:a.out::return 
/ uregs[R_EIP] == 0x40404040 / { 
prainti(”’I’m-qoing to @rash 111%); 
printf (“Module: %s Function %s”,probemod, probefunc) ; 
G@lustack (10) |=count()+ he, 
} 


10 deep userland stack 


Here is where r_ Erp constant came from: 
uregs[] Array 


The uregs[{] array enables you to access individual us- 
er registers. The following tables list indices into the 
uregs[] afray corresponding to each supported Solaris 
system architecture. On AMD64 platforms, the uregs ar- 


Table 6. Common uregs[] Constants 


R_PC program counter register 
R_SP stack pointer register 
R_RO first return code 

R_R1 second return code 
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¢  http://dtrace.org/ 

¢ https://wikis.oracle.com/display/DTrace 

« — http://bsdmag.org/magazine/1800-bsd-security-protect-your-bsd 


ray has the same content as it does on x86 platforms, 
plus the additional elements listed in Table 5. The aliases 
listed in Table 6 can be used on all platforms. 


Second Case Scenario 

You want to take a look at every string that is being writ- 
ten, as you have encountered that a file that has been cor- 
rupted by the word “COW”. 


syscall::write:entry 
{ 
if(copyinstr(argl) == “COW”) 
{ 
printf(“ some one wrote COW “); 


ustack(); //--> check user stack 


Third Case Scenario 
Let’s check malloc return pointer and size requested. Nice 
for quick debugging 


pidStarget::malloc:entry{ 
self->trace = 1; 
self->size = arg0; 
} 
pidStarget::malloc:return 
/self->trace == 1/ 
{ 
ustack (1); 
printf (“malloc return: <ptr=0x%p> <size=%d>”", argl, self->size); 
self->trace = 0; 


self->size = 0; 


Hope this was as useful for you as it was for me! Now it’s 
just a matter of really what you want to look at with DTrace. 


CARLOS ANTONIO NEIRA 

Carlos Antonio Neira is a C, Unix and Mainframe developer. He de- 
velops in asm and does some kernel development for a living. In 
his free time he contributes to open source projects. Apart from 
that, he spends his time on testing and experimenting with his ma- 
chines. What gives hima a lot of enjoyment is solving old problems 
with new ideas. You may reach him at: cneirabustos@gmail.com. 
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Please join us October 25-27, 2013 at the Hyatt in Dulles, Virginia for the first biennial 
vBSDCon event. This exciting weekend will bring together members of the BSD community for a 
series of roundtable discussions, educational sessions, best practice conversations, and exclusive 
networking opportunities. See below for details on this industry weekend not to be missed: 


AGENDA 
¢ Friday, October 25: Evening Reception 
- Saturday, October 26: General Session, Birds of a Feather Sessions 
¢ Sunday, October 27: General Session, Breakout Sessions 


WHO SHOULD ATTEND 
-Developers «Engineers «Administrators + Innovators 


TOPICS 
« PkKgNG w/ Baptiste Daroussin 
- A comprehensive look at bsdinstall with Devin Teske 
- Netflix Demo/Presentation with Scott Long 
«netmap with Luigi Rizzo 
« Migration from GCC to LLVM/Clang with David Chisnall 


REGISTRATION INFORMATION 
WILL BE SENT TO YOU IN MAY! 


Questions? Please contact: eventsteam @verisign.com 
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